Papers with multimodal machine learning
Tutorial on Multimodal Machine Learning (2022.naacl-tutorials)
Copied to clipboard
| Challenge: | Multimodal machine learning is a challenging but crucial area with numerous applications in multimedia, affective computing, robotics, finance, HCI, and healthcare. |
| Approach: | This tutorial will describe an updated taxonomy on multimodal machine learning synthesizing its core technical challenges and major directions for future research. |
| Outcome: | The proposed taxonomy synthesizes the core technical challenges and major directions for future research. |
Does my multimodal model learn cross-modal interactions? It’s harder to tell than you might think! (2020.emnlp-main)
Copied to clipboard
| Challenge: | a new tool for evaluating expressive cross-modal interactions is needed . empirical multimodally-additive function projection is a tool for isolating unimodal structure . |
| Approach: | They propose a tool that modifies model predictions so that cross-modal interactions are eliminated . they propose to use EMAP to evaluate models' ability to leverage cross-module interactions . |
| Outcome: | The proposed tool can be used to evaluate models on image+text classification tasks . it finds that removing cross-modal interactions results in little to no performance degradation . |
Modality-specific Learning Rates for Effective Multimodal Additive Late-fusion (2022.findings-acl)
Copied to clipboard
| Challenge: | Multimodal machine learning uses additive late-fusion to combine feature representations from different modalities into a joint representation. |
| Approach: | They propose a Modality-Specific Learning Rate method to build late-fusion multimodal models from fine-tuned unimodal models. |
| Outcome: | The proposed method outperforms global learning rates on multiple tasks and settings and enables the models to effectively learn each modality. |
Dynamic Regularization in UDA for Transformers in Multimodal Classification (2023.acl-long)
Copied to clipboard
| Challenge: | Multimodal machine learning is a cutting-edge field that explores ways to combine information from multiple sources into models. |
| Approach: | They propose a multimodal BERT-ViT model that exploits weaker modality while regularizing the loss function. |
| Outcome: | The proposed model exploits weaker modality while regularizing the loss function. |